An Asynchronous Parallel Stochastic Coordinate Descent Algorithm
نویسندگان
چکیده
We describe an asynchronous parallel stochastic coordinate descent algorithm for minimizing smooth unconstrained or separably constrained functions. The method achieves a linear convergence rate on functions that satisfy an essential strong convexity property and a sublinear rate (1/K) on general convex functions. Near-linear speedup on a multicore system can be expected if the number of processors is O(n) in unconstrained optimization and O(n) in the separable-constrained case, where n is the number of variables. We describe results from implementation on 40-core processors.
منابع مشابه
Asynchronous Doubly Stochastic Proximal Optimization with Variance Reduction
In the big data era, both of the sample size and dimension could be huge at the same time. Asynchronous parallel technology was recently proposed to handle the big data. Specifically, asynchronous stochastic (variance reduction) gradient descent algorithms were recently proposed to scale the sample size, and asynchronous stochastic coordinate descent algorithms were proposed to scale the dimens...
متن کاملAsynchronous Parallel Greedy Coordinate Descent
In this paper, we propose and study an Asynchronous parallel Greedy Coordinate Descent (Asy-GCD) algorithm for minimizing a smooth function with bounded constraints. At each iteration, workers asynchronously conduct greedy coordinate descent updates on a block of variables. In the first part of the paper, we analyze the theoretical behavior of Asy-GCD and prove a linear convergence rate. In the...
متن کاملPASSCoDe: Parallel ASynchronous Stochastic dual Co-ordinate Descent
Stochastic Dual Coordinate Descent (DCD) is one of the most efficient ways to solve the family of `2-regularized empirical risk minimization problems, including linear SVM, logistic regression, and many others. The vanilla implementation of DCD is quite slow; however, by maintaining primal variables while updating dual variables, the time complexity of DCD can be significantly reduced. Such a s...
متن کاملAsynchronous Accelerated Stochastic Gradient Descent
Stochastic gradient descent (SGD) is a widely used optimization algorithm in machine learning. In order to accelerate the convergence of SGD, a few advanced techniques have been developed in recent years, including variance reduction, stochastic coordinate sampling, and Nesterov’s acceleration method. Furthermore, in order to improve the training speed and/or leverage larger-scale training data...
متن کاملPerturbed Iterate Analysis for Asynchronous Stochastic Optimization
We introduce and analyze stochastic optimization methods where the input to each gradient updateis perturbed by bounded noise. We show that this framework forms the basis of a unified approachto analyze asynchronous implementations of stochastic optimization algorithms. In this framework,asynchronous stochastic optimization algorithms can be thought of as serial methods operatin...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2014